WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) Internationa) Patent Classification 6 : 
. G06T 5/20 



Al 



(11) International Publication Number: 
(43) International Publication Date: 



WO 99/17254 

8 April 1999 (08.04.99) 



(21) International Application Number: PCT/US98/1 9428 

(22) International Filing Date: 17 September 1998 (17.09.98) 



(30) Priority Data: 

08/938,384 



26 September 1 997 (26.09.97) US 



(81) Designated States: JP, European patent (AT, BE, CH, CY, DE, 
DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE). 



Published 

With international search report. 



(71) Applicant: POLAROID CORPORATION [US/US]; 784 

Memorial Drive, Cambridge, MA 02139-3589 (US). 

(72) Inventors: BOUCHER, Ronald, E.; 8408 Cedar Street, Silver 

Spring, MD 20910 (US). DEBOSSU, John, C; 103 
Marbury Avenue, Pawtucket, RI 02860 (US). 

(74) Agent: MOFFORD, Donald, F.; Polaroid Corporation, 784 
Memorial Drive, Cambridge, MA 02139-3589 (US). 



(54) Tide: DIGITAL REDEYE REMOVAL 




(57) Abstract 

A system and method for correcting redeye in a digital image includes the functions or steps, respectively, of: presetting threshold 
values of luminance, hue and saturation; selecting at least one pixel of the image for comparison with the threshold values; identifying redeye 
pixels of the image by comparing luminance, hue and saturation values which are derived from color components of the selected pixels with 
the threshold values; indentifying redeye regions of the image having one or more adjacent redeye pixels which are surrounded by non-redeye 
pixels; replacing the luminance, hue and saturation values of the identified redeye pixels with values derived from weighted functions of 
the color components subject to a first set of color weights; identifying edge pixels as the redeye pixels which border non-redeye pixels; 
and smoothing each redeye region by replacing the luminance, hue and saturation values of the edge pixels with the weighted functions of 
the color components subject to a second set of color weights. 
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DIGITAL REDEYE REMOVAL 

R A P.K("?R OI JND OF THF. INVENTION 

In some color and black-and-white pictures taken by an electronic flash the eyes of a human 
or an animal have an unnatural "redeye" effect which causes pupils to appear red rather than black, 
brown, blue, green or gray. It is generally accepted that the redeye effect is caused by light from 
the electronic flash that is reflected from the retina of the subject's eye. Thus, light from the flash is 
imaged at the retina where a fraction of it is reflected back out the pupil of the eye. If the eye were 
a perfect optical system, this light would be reflected entirely back into the flash source. However, 
since the eye is not perfect, and since the original path of the light originating at the electronic flash 
is proximate to the lens axis, then some of the spreading reflected light from the retina overlaps the 
lens axis and is sensed by the lens. Since the reflectivity of the retina is high in the red light portion 
of the visible light spectrum, the pupil spot effected by the reflected light is highly red. Although 
significant subject variations in susceptibility to redeye have been noted,-these can be explained by 
variability in retinal-reflectance, accommodation, pupil size and ocular optics of the subjects. 

The redeye effect can be avoided by locating the flash unit some distance (perhaps six 
inches or more) from the lens axis, viewing the subject from an angle, having the subject look away 
from the camera, or raising the ambient light level so that the pupils will close to a significant 
degree. Many other methods and systems have been utilized for ininimizing redeye such as 
reducing the size of the pupil by sending a pulsating light from the electronic flash just prior to 
taking the picture. 

It is a primary object of the present invention to provide a system and method for redeye 
removal in a digital image. 
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SUMMARY OF THE INVENTION 

A system and method for correcting redeye in a digital image includes the functions or 
steps, respectively, of: presetting threshold values of luminance, hue and saturation; selecting at 
least one pixel of the image for comparison with the threshold values; identifying redeye pixels of 
5 the image by comparing luminance, hue and saturation values which are derived from color 

components of the selected pixels with the threshold values; identifying redeye regions of the image 
having one or more adjacent redeye pixels which are surrounded by non-redeye pixels; replacing 
the luminance, hue and saturation values of the identified redeye pixels with values derived from 
weighted functions of the color components subject to a first set of color weights; identifying edge 
1 0 pixels as the redeye pixels which border non-redeye pixels; and smoothing each redeye region by 
replacing the luminance, hue and saturation values of the edge pixels with the weighted functions of 
the color components subject to a second set of color weights. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The aforementioned aspects and other features of the invention are described in detail in 
1 5 conjunction with the accompanying drawings where: 

Figure 1 is a diagram of an image acquisition and rendering system including an image 
processing system constructed in accordance with the principles of the invention; 

Figure 2 is a block diagram of a preferred sequence of steps of the inventive method; and 
Figure 3 is a representation of a redeye region composed of redeye pixels. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

A digital image consists of pixels each containing color and luminance information which 
can be represented in a variety of domains. For instance, the pixel information can be resident in 
RGB (red, green and blue) device-dependent color space having R, G and B color components for 
5 use with a color monitor, or in CMYK (cyan, magenta, yellow black) device-dependent color space 
having C, M, Y and K color components for use with a color printer. When transferring 
information between such device-dependent color spaces, use is made of device-independent color 
space such as, but not limited to, proprietary color spaces, CIE L*a*b*, CIE XYZ, YUV and YIQ. 

In one embodiment of the redeye removal method, the properties of a pixel are represented 
10 in RGB color space and the pixel values are transformed into YIQ color space as defined in the 
following equations (1) through (5) where, for each pixel, Y represents luminance, I and Q 
represent chrominance, H represents hue and S represents saturation. 



Y = 0.30R + 0.59G + 0.11B (1) 

I = 0.5R-0.23G-0.27B (2) 

15 Q = 0.203R - 0.5G + 0.297B (3) 

H = tan- , (Q/I) (4) 

S^ + Q 2 } 172 (5) 



The transformations of equations (1) through (5) are conventional as described in "Colour Space 
In Television And Display Systems" by W. N. Sposon, Adam Hilger Ltd., Bristol (1983), pages 19 
20 and 64. 

Maximum and minimum threshold values of luminance, hue and saturation for defining a 
redeye region of a pixel are determined in block 30 of Figure 2 by testing a database of images 
where: MaxLum represents the maximum luminance threshold value; MinLum represents the 
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minimum luminance threshold value; MaxHue represents the maximum hue threshold value; 
MinHue represents the minimum hue threshold value; MaxSat represents the maximum saturation 
threshold value; and MinSat represents the minimum saturation threshold value. Threshold ranges 
having both upper and lower limits are necessary for preventing spurious redeye readings. These 
threshold values, the calculation of which will be detailed further on, are preferably precalculated 
and prestored in a configuration file, database, look-up table or other memory structure prior to 
initiation of the redeye removal method. Furthermore, the above threshold values may be varied if 
desired by the user. 

One or more pixels of the image are selected in block 32 for comparison against the 
predetermined threshold values, MaxLum, MinLum, MaxHue, MinHue, MaxSat and MinSat. The 
comparison starts with one seed pixel and continues in some predetermined pattern until the 
luminance, hue and saturation properties of all pixels for a given area (which could be a part, or the 
whole, of an image) are compared against the predetermined threshold values. If each of the 
luminance, hue and saturation values of a pixel being tested is both greater than the respective 
minimum threshold value and less than the respective maximum threshold value, then that pixel in 
block 34 is identified and labeled as a redeye pixel, i.e. a pixel which exhibits the redeye effect. For 
instance, a pixel at a spatial location (x,y) could be flagged or labeled as a redeye pixel by setting 
the appropriate element in the array Flags(x,y) equal to 3 whereas a non-redeye pixel could be 
indicated by setting Flags(x,y) equal to 0. 

Redeye regions are identified in block 36 as regions of the image having one or more 
adjacent redeye pixels which are surrounded by non-redeye pixels. After all redeye pixels and 
redeye regions have been identified in block 36, then the original luminance, hue and saturation 
values of those pixels are replaced in block 38 with values of luminance, hue and saturation derived 
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from weighted functions of the color components. The weighted functions of the color 
components are determined in equations (7), (8) and (9), and the replacement luminance, hue and 
saturation values are derived by inserting the weighted functions R', G' and B' into equations (1), 
(4) and (5). The density of each pixel after insertion of replacement values is typically dark neutral 
gray to closely resemble a pupil. Other Y, H and S replacement values corresponding to! different 
shades of gray, or different eye colors such as blue, green, brown or gray could be preselected by 
the user if desired. 

In order to provide a more natural effect for the resulting pupils which have had redeye 
removed, the edges of the redeye regions are smoothed or feathered so that the colors of the pupil 
and the retina will gradually blend together. This is accomplished by first setting each redeye pixel 
of each redeye region to dark neutral gray as noted above, then blending the tones of edge pixels of 
the redeye region from dark neutral gray to a lighter gray to account for flare and specular 
reflection. 

The edge pixels are denned and identified in block 40 as redeye pixels in a redeye region 
which border non-redeye pixels. Figure 3 illustrates an 8x8 block 50 of redeye pixels, Le. a redeye 
region 50, where each pixel 1 is an edge pixel. Thus feathering or smoothing of redeye region 50 in 
the simplest form amounts to setting all pixels 1, 2, 3 and 4 to dark neutral gray, and then replacing 
the edge pixels 1 with an intermediate gray value. However, the extent of feathering can be 
extended beyond-one layer of edge pixels. This is accomplished in block 42 by flagging layers of 
edge pixels. Each pixel is tested to determine if it should be designated as a redeye pixel, and if so, 
then the particular tone or level of gray which should be assigned to each layer of pixels is 
determined by the amount of feathering desired. For instance in Figure 3, the redeye region 50 is 
surrounded by non-redeye pixels (not shown) at locations (x,y) which are flagged as Flags(x,y) = 0. 
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All redeye pixels 1, 2, 3 and 4 are initially flagged as Flaggy) = 3 corresponding to pixels to be 

replaced with dark neutral gray. Thereafter, pixels 3 and 4 remain flagged as Flags(x,y) = 3 

corresponding to dark neutral gray for producing the most correction, pixels 2 are flagged as 

Flags(x,y) = 2 corresponding to an intermediate gray for producing medium correction, and pixels 

1 are flagged as Flags(x,y) = 1 corresponding to a light gray for producing a small correction. Each 

set of pixels designated by a different Flags value will have its original luminance, hue and 

saturation values replaced. This feathering, as exemplified by the following pseudocode, produces 

a more natural, gradual blend between the colors of the pupil and the retina. 

for each x,y in oldBorder 
for xl,yl = each neighbor pixel of x,y and Flagsfrl.yl) = 0 
add xl.yl to newBorder 
setFlags(xl,yl) = 2 

swap oldBorder and newBorder pointers 
for each x,y in oldBorder 

15 forxl,yl= each neighbor pixel of x,y and Flags(xl yl) = 0 

setFlags(xl ) yl)=l 

Equations (6) through (9) calculate weighted functions R', G', and B' from the R, G and B 

color components which, in turn, are inserted into equations (1) through (5) to derive replacement 

20 luminance, hue and saturation values for any pixel which is identified as a redeye pixel. 

C = min(RG,B) (6) 

R'={C*n+R*(4-n)}/4 (7) 

G'={C*n + G*(4-n)}/4 (8) 

B'={C*n + B*(4-n)}/4 (9) 

25 where: R\ G' and B' are the weighted functions of the color components corresponding to 

feathered colorant values which replace the original values R G and B; and n is a variable defined 

as the color weighting value so that n = 4 for Eags = 3, n = 2 for Flags = 2, and n = 1 for 
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Flags = 1 . The red color component R, the blue color component B and the green color 
component G are compared with one another and the numerical minimum value of the three color 
components is set to C, which is then applied to equations (7), (8) and (9). 

Three modes are discussed for locating or selecting pixels in the image for comparison with 
5 the threshold values. They are the unassisted mode, the partially assisted mode and the fUlly 
assisted mode. 

The predetermined threshold values include a maximum luminance threshold value 
MaxLum, a minimum luminance threshold value MinLum, a maximum hue threshold value 
MaxHue, a minimum hue threshold value MinHue, a maximum saturation threshold value MaxSat, 
10 and a minimum saturation threshold value MinSat. In the unassisted mode, for a given pixel having 
a luminance Y, a hue H and a saturation S to be designated as a redeye pixel, the conditions of 
equations (10), (1 1) and (12) must be met. 

MinLum < Y < MaxLum (10) 
' MinHue <H<MaxHue (11) 
15 MinSat £S<MaxSat (12) 

In other words, the predetermined threshold values establish the redeye range for each of the 
designated pixel properties. 

Some of the pixels identified as potential redeye pixels could actually be non-redeye pixels 
which exhibit the same over-threshold attributes but which are located in parts of the image away 
2 0 from the pupils. For instance, a bright red shirt could possibly result in erroneous redeye pixel 
findings. In order to minimize the occurrence of this type of error, additional threshold values are 
applied on the size and shape of the potential redeye regions. Since a pupil has an aspect ratio of 
approximately 1:1, then any potential redeye region having an aspect ratio greater than, say, 2: 1 or 
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1 :2 would be dismissed as a non-redeye region. Although the unassisted mode identifies all 
potential redeye pixels, it is relatively slow, computationally extensive and prone to false redeye 
pixel identification. 

In the fully assisted mode, the coordinates of the seed pixels of each region are supplied by 

the user via a data structure input to the algorithm. For instance, the user will move the cursor to a 

redeye region in the image, then click the cursor on the redeye region with the mouse. The 

predetermined threshold values for the fully assisted mode are represented as: 

MinLum = Y - DeltaLumM (13) 

MaxLum = Y + DeltaLumP (14) 

MinHue = H - DeltaHueM ( 1 5) 

MaxHue = H + DeltaHueP (16) 

MinSat = S - DeltaSatM (17) 

MaxSat = S + DeltaSatP (18) 

where as before Y, H and S are, respectively, the luminance, hue and saturation values of the 

selected-pixels. The values DeltaLumM, DeltaLumP, DeltaHueM, DeltaHueP, DeltaSatM and 

DeltaSatP are preselected by testing images from a database. 

After the threshold values have been preset as previously described, then a number of 

points are selected by the user as seed pixels for suspected redeye regions. Although the maximum 

number of selected points is theoretically limited by the number of pixels in the image, the practical 

limitation is in the order of 10 to 20. As the number of selected points increases, the computational 

time and processing requirements also increase, so it is preferred to select as few points as possible 

for identification of redeye. For instance, in an image having two people with redeye, a minimum 

number of 4 points is preferably selected, one for each eye. 
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MISSING UPON TIME OF PUBLICATION 



AUSENTE EN EL MOMENTO DE LA PUBLICACION 



MANQUANT AU MOMENT DE LA PUBLICATION 
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In the partially assisted mode, the user clicks the mouse cursor near the corner of an eye 
then pulls the cursor away from that point to create a rectangle around the area of interest where 
the user identifies redeye. Upon selection by the user of a redeye removal button on the screen, the 
method as described herein automatically identifies and replaces any luminance, hue and saturation 
values of redeye pixels within the selected rectangular region. However, for the partially assisted 
mode the saturation criterion is augmented by a function based on the color of the pixel. The seed 
pixel is determined as the pixel within the selected rectangle which maximizes the expression (19): 

S + HueFactor * (HueRedMax - abs(H - HueRed))/HueRedMax (19) 
Only the pixels located within the user-specified rectangle are tested for redeye, so computational 
efficiency and processing times are minimized. The above steps can be repeated as desired for 
additional redeye regions with the image. 

Figure 1 is a diagram of an image acquisition and rendering system 26 built in 
accordance with the principles of the invention. The system 26 includes a computer 14, one or 
more image acquisition devices such as the camera 12 and the scanner 10 which are useful for 
acquiring an image and converting it to digital image data, and one or more image rendering 
devices such as the monitor 16 and the printer 20 which are useful for generating an output 
image. 

The system 26 includes conventional operator interface elements such as the keyboard 
18, the mouse 22, the monitor 16 and a compact or floppy disk 24. During operation of the 
system 26, the computer 14 processes certain programs which enable selected image 
processing operations such as the redeye removal algorithm described hereinbefore. The 
system 26 may include a variety of components in addition to or instead of those depicted in 
Figure 1. For instance, the various elements of the system 26 may be replaced by 

10 
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special-purpose hardware, firmware or any combination thereof. Also, those skilled in the art 
understand that many, if not all, of the processing capabilities of the computer 14 can 
alternatively be performed in intelligent system components, such as the camera 12, the 
scanner 10, the monitor 16, the keyboard 18 and the printer 20. 
5 The redeye removal algorithm, as described above, is loaded into the computer 14, for 

instance, via a CDROM 24 and then run by the user who interacts with the computer 14 by 
way of the keyboard 18 and the monitor 16. First, an image is acquired, for instance, through 
an image acquisition device 10 or 12, or from a file stored within the computer 14. Means for 
selecting one or more pixels for comparison with the threshold values is defined as the 

1 0 computer 14 when the algorithm is operating in the unassisted mode, or as the mouse 22 when 
the algorithm is operating in either the assisted and partially assisted modes. The computer 14 
then performs the functions of identifying redeye pixels, identifying redeye regions, replacing 
redeye pixel values, identifying edge pixels and the smoothing. 

• The above embodiments of the present invention are merely illustrative of the many 

15 possible specific embodiments and variations of the invention as claimed. Many other 
equivalents of the claimed devices and methods are within the purview of the claims as 
understood by those of ordinary skill in the art. 
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CLAIMS: 

1 . A method for correcting redeye in a digital image, comprising the steps of: 

presetting threshold values of luminance, hue and saturation; 

selecting at least one pixel of the image for comparison with said threshold values; 

identifying redeye pixels of the image by comparing luminance, hue and saturation values 
5 which are derived from color components of said selected pixels with said threshold values; 

identifying redeye regions of the image having one or more adjacent redeye pixels which 
are surrounded by non-redeye pixels; 

replacing the luminance, hue and saturation values of the identified redeye pixels with 
values derived from weighted functions of the color components; 
1 0 identifying edge pixels as said redeye pixels which border non-redeye pixels; and 

smoothing each said redeye region by replacing the luminance, hue and saturation values of 
the edge pixels with the weighted functions of the color components subject to a second set of 
color weights. 

2. The method of claim 1 wherein said threshold values of luminance, hue and saturation 
are determined by testing on a database of images. 

3. A system for correcting redeye in a digital image, comprising: 

means for selecting at least one pixel of the image for comparison with preset threshold 
values of luminance, hue and saturation; 

12 
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means for identifying redeye pixels of the image by comparing luminance, hue and 
saturation values which are derived from color components of said selected pixels with said 
threshold values; 

means for identifying redeye regions of the image having one or more adjacent redeye 
pixels which are surrounded by non-redeye pixels; 

means for replacing the luminance, hue and saturation values of the identified redeye pixels 
with values derived from weighted functions of the color components subject to a first set of color 
weights; 

means for identifying edge pixels as said redeye pixels which border non-redeye pixels; and 
means for smoothing each said redeye region by replacing the luminance, hue and 

saturation values of the edge pixels with the weighted functions of the color components subject to 

a second set of color weights. 

4. The system of claim 3 wherein said threshold values of luminance, hue and saturation 
are determined by testing on a database of images. 
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